Image processing apparatus and image pickup apparatus

ABSTRACT

An image processing apparatus includes a memory configured to store information of an OTF or a PSF of an optical system for at least one of capturing conditions, and an image processor configured to generate secondary and higher components of a phase of an OTF or a shape component of a PSF corresponding to a capturing condition of a captured image through an interpolation based on at least two OTFs or PSF in the memory, while center-of-gravity positions or maximum intensity positions are accorded with each other or differential root-mean-square values of the point spread functions are minimum, and to restore the image utilizing the generated OTF or an OTF derived from the generated PSF.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus and animage pickup apparatus, which restore an (deteriorated) image that hasbeen deteriorated by an aberration of an image pickup system, to an(original) image or pre-deterioration image.

2. Description of the Related Art

One conventionally known technology is to restore an image deterioratedby an aberration of an optical system (referred to as “image restorationprocessing” hereinafter). One image restoration processing a method forusing information of an optical transfer function (“OTF”) or a pointspread function (“PSF”) of an optical system which has a Fouriertransform relationship with the OTF.

The OTF has a real part and an imaginary part, and is generally storedas two-dimensional data in storage, such as a memory. Thistwo-dimensional data will be referred to as “OTF data” hereinafter. Inthe general image restoration processing, OTF data is prepared for eachRGB, and the OTF data for one image height is a tap number in the xdirection×a tap number in the y direction×2 (real part, imaginarypart)×3 (chromatic components). The OTF and PSF are different accordingto capturing conditions, such as an image height of an image capturedvia the optical system, and a focal length, an F-value, and an objectdistance of the optical system.

Japanese Patent Laid-Open No. (“JP”) 2005-308490 proposes a method forinterpolating an optical characteristic of glasses having an arbitraryobject distance, and JP 2003-132351 assumes an elliptical PSF, and amethod for generating a PSF through an interpolation according to animage height position.

The above methods of storing OTF data for each chromatic component andfor at least one of capturing conditions cause a data amount to beenormous. Accordingly, this inventor attempts to store discrete OTF datacorresponding to representative capturing conditions in an imageprocessing apparatus (or image pickup apparatus), and to generate OTFdata corresponding to remaining capturing conditions through aninterpolation utilizing stored OTF data. At this time, there is atrade-off relationship between a reduced data amount of the OTF data andthe interpolation accuracy (image restoring precision).

However, when PSF or OTF data are interpolated while thecenter-of-gravity positions of the PSFs used for the interpolation arediscarded with each other (or primary components of the phases of theOTFs are discarded with each other), the interpolation cannot be highlyprecise. Prior art is silent about a solution for this problem.

SUMMARY OF THE INVENTION

The present invention provides an image processing apparatus and animage pickup apparatus, which can restrain OTF data capacity to bestored, and perform a highly precise image restoration.

Although the above conventional problem discusses the OTF, the PSF has aFourier transform relationship with the OTF and thus the same problemoccurs when the PSF data is stored and used for the image restorationprocessing.

An image processing apparatus or an image pickup apparatus according tothe present invention includes a memory configured to store informationof an optical transfer function or a point spread function of an imagepickup optical system for at least one of capturing conditions, and animage processor configured to generate secondary and higher componentsof a phase of an optical transfer function or a shape component of apoint spread function corresponding to a capturing condition of an imagecaptured through the image pickup optical system through aninterpolation based on at least two optical transfer functions or pointspread functions which correspond to different capturing conditions andare derived from the information stored in the memory, whilecenter-of-gravity positions or maximum intensity positions are accordedwith each other or differential root-mean-square values of the pointspread functions are minimum, and to restore the image utilizing anoptical transfer function derived from the optical transfer functionthat has been generated through the interpolation or an optical transferfunction derived from the point spread function that has been generatedthrough the interpolation.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram of an image processing apparatus and FIG. 1Bis a flowchart of image restoration processing according to a firstembodiment.

FIGS. 2A and 2B are views for explaining a production example of datastored in a memory illustrated in FIG. 1A according to the firstembodiment.

FIGS. 3A and 3B are views for explaining S16 illustrated in FIG. 1Baccording to the first embodiment.

FIG. 4 is a block diagram of a digital camera according to a secondembodiment.

DESCRIPTION OF THE EMBODIMENTS

In the image restoration, the following expression is established where(x, y) is a real space, f(x, y) is an original image before it isdeteriorated by an optical system, h(x, y) is a PSF, and g(x, y) is adeteriorated image.g(x,y)=∫∫f(X,Y)·h(x−X,y−Y)dXdY  (1)

The following expression is established by Fourier-transformingExpression (1) to convert the real space (x, y) into the frequency space(u, v) where F(u, v) is a Fourier transform of f(x, y), G(u, v) is aFourier transform of g(x, y), and H(u, v) is a Fourier transform of h(x,y) and an optical transfer function (“OTF”):G(u,v)=F(u,v)·H(u,v)  (2)

The following expression is established from Expressions (1) and (2):F(u,v)=G(u,v)/H(u,v)  (3)

Therefore, F(u, v) can be obtained by dividing the Fourier transformG(u, v) by H(u, v) in the frequency space, and the original image f(x,y) can be obtained by inverse-Fourier-transforming F(u, v).

Since the above processing actually amplifies noise, it is known thatthe following Wiener filter may be used for 1/H(u, v) in Expression 3where Γ is a constant used to reduce an amplification amount of thenoise:1/H(u,v)≡1/H(u,v)·|H(u,v)|²/(|H(u,v)|²+Γ)  (4)

Multiplying the OTF having a frequency and phase information of theoptical system by Expression (4) can make zero a phase of the PSF causedby the diffraction and aberration of the optical system, amplifies thefrequency characteristic of the amplitude, and provide a highly preciseand well restored image.

Although it is thus necessary to obtain precise OTF information of theimage pickup optical system, an optical performance, such as an F-valueand aberration, of the image pickup optical system used for a cameragenerally significantly fluctuates among image heights. In order tocorrect the deterioration of the object image, Expression (4) cannot beused for batch calculating in the frequency space as it is, andExpression (4) is converted into an (image restoring) filter in the realspace for each image height.

In the image restoration utilizing the image restoring filter, it isnecessary to store the image restoring filter or the OTF information forproducing the image restoring filter in the apparatus. When informationof the image restoring filter is stored in the image pickup apparatus,the calculation used to correct the deterioration of the image is onlyfiltering processing, and the processing becomes faster. However, achange of the image restoring filter becomes impossible, and it isdifficult to control the level of the deteriorated correction.

On the other hand, if the OTF information used to produce the imagerestring filter is stored, the level of the deteriorated correction canbe freely controlled in accordance with the deterioration degree of theobject image. The image restoring filter can be generated byFourier-transforming the OTF information.

The OTF is two-dimensional data having a real part and an imaginarypart. In the general image restoration, the wavelength is expressed byvariables of three chromatic components of RGB, and thus the OTF data ofone image height is expressed by a tap number in the x direction×a tapnumber in the y direction×2 (real part, imaginary part)×3 (chromaticcomponents). In addition, the OTF is different according to a capturingcondition, such as an image height, a focal length, an F-value, and anobject distance. Therefore, storing the OTF data for each chromaticcomponent and for at least one of capturing conditions causes a dataamount to be impractically enormous.

Accordingly, discrete OTF data corresponding to representative capturingconditions are stored in the image processing apparatus (or image pickupapparatus), and OTF data corresponding to remaining capturing conditionsare generated through an interpolation utilizing the stored OTF data. Atthis time, there is a trade-off relationship between a reduced dataamount of the OTF data and the interpolation accuracy (image restoringprecision).

With respect to an absolute amount of the aberration in the generalimage pickup optical system, a positional shift amount of an image pointfor each wavelength caused by the lateral chromatic aberration among thewavelengths is larger than the spread of the PSF and becomesparticularly remarkable at an off-axis object point. This corresponds toa primary component of the phase of the OTF. The phase component of theOTF is a tan(Im(OTF)/Re(OTF)) and the primary component will be referredto as a primary phase.

However, the center-of-gravity position of the PSF (point image) differsaccording to the capturing condition even for the same wavelength. Forexample, when a PSF at a target position that is located between a firstposition, such as (0, a), and a second position, such as (0, b), in theXY coordinate system is generated by performing a linear interpolationutilizing the PSF that has a center-of-gravity position at the firstposition, and the PSF that has a center-of-gravity position at thesecond position, a shape of the interpolated PSF destroys.

Accordingly, in highly precisely interpolating the shape of the PSF,this embodiment accords (aligns or matches) two center-of-gravitypositions (for example, by moving them to the origin) and interpolatesthe shape through weighting according to these two center-of-gravitypositions. In other words, the primary component of the phase of the OTFwhich represents the center-of-gravity position of the PSF is eliminatedin the frequency space, and uses secondary and higher components for theinterpolation.

First Embodiment

FIG. 1A is a block diagram of the image processing apparatus 20according to a first embodiment. FIG. 1B is a flowchart of imagerestoration processing executed by the image processing apparatus 20according to the first embodiment, and “S” stands for the step. An imageprocessing method (image processing program) illustrated in FIG. 1B actsto instruct a computer to serve each step.

The image processing apparatus 20 is a separate unit from a camera(image pickup apparatus) 10 in FIG. 1A, but may be integrated with thecamera 10 as described later. The camera 10 generates an object image(deteriorated image) that has been deteriorated by an aberration of animage pickup optical system. The object image is generated as a resultof that an image pickup element photoelectrically converts an opticalimage captured by the image pickup optical system.

The image processing apparatus 20 includes an image processing/operatingunit 22 and a memory 24, and may include a computer and software (imageprocessing program) installed in the computer.

The image processing/operating unit 22 is an image processor configuredto provide image processing including image restoration processing, andincludes a microcomputer (processor). The image processing/operatingunit 22 can restore an image by utilizing information of the opticaltransfer function (“OTF”) or the point spread function (“PSF”) stored inthe memory 24.

The memory 24 stores a program that contains image processing includingimage restoration processing, and information of the OTF or PSF of theimage pickup optical system of the camera, for at least one of capturingconditions, such as an image height, a focal length, an F-value, and anobject distance. This embodiment allows a capturing condition other thana combination of the image height, the focal length, the F-value, andthe object distance. Since the memory 24 merely stores the informationof the OTF or PSF for a part of the combination of the capturingconditions, the storage capacity can be reduced.

According to this embodiment, the information of the OTF (OTF data) orthe PSF stored in the memory 24 is information of secondary and highercomponents from which a primary component of the phase of the OTF iseliminated or information of a shape component from which acenter-of-gravity position component of the PSF is eliminated. However,the OTF or PSF information stored in the memory 24 may contain theprimary component of the phase of the OTF or the center-of-gravity ofthe PSF. In this case, the image processing/operating unit 22 uses thisinformation to generate pre-interpolation information of the secondaryand higher components from which a primary component of the phase of theOTF is eliminated or information of a shape component from which acenter-of-gravity position component of the PSF is eliminated.

FIGS. 2A and 2B are views for explaining a production example of thedata stored in the memory 24. The OTF is a frequency response of the PSFcalculated by the Fourier transform of the PSF, and the PSF and the OTFpossess equivalent information. A method of obtaining the PSF mayinclude calculating a wavefront aberration of the optical system,generating a pupil function, Fourier-transforming the pupil function,and squaring the resultant absolute value.

The way of selecting the reference wavelength and the referencespherical surface is arbitrary in calculating the wavefront aberration.For example, the acquired PSF is different between the calculationaround the ideal image height determined by the paraxial magnificationof the optical system and the calculation around a terminus of theprincipal ray obtained through ray tracing of the actual optical system.

According to the former calculation, the PSF contains a distortioncomponent of the reference wavelength, and thus the entire PSF shifts bythe distortion amount. According to the latter calculation, the PSFcontains no distortion component of the reference wavelength, and thePSF does not shift.

This embodiment may arbitrarily sets the center of the referencespherical surface when the PSF is calculated, but if the center of thereference spherical surface is not set to the terminus of the principalray, the generated distortion component of the reference wavelength iseliminated for the calculated wavelength after the PSF is calculated.

A description will be given of a generation of the PSF through aninterpolation which has an intermediate focal length in one illustrativezooming optical system, when only the focal length is different betweenthe two PSFs. The PSF interpolations between the image heights, betweenthe F-values, and between the object distances may utilize similarapproaches.

The PSF used for the interpolation may correspond to a single wavelengthbut may correspond to a plurality of wavelengths when the wavelengthsare weighted according to the spectrum intensity distribution of thearbitrary light source and added up.

Initially, the image processing/operating unit 22 selects a PSFcorresponding to a first capturing condition (h1, f1, F1, d1) and a PSFcorresponding to a second capturing condition (h1, f2, F1, d1) as twodifferent PSF data used for the interpolation. In the meanwhile, it issufficient that at least two PSFs or OTFs corresponding to differentcapturing conditions are used for the interpolation. The generated PSFdata through the interpolation is set to a PSF corresponding to acapturing condition (hi, fj, Fk, dl) which has an image height hi, afocal length fj, an F-value Fk, and an object distance dl.

The lateral chromatic aberration component is different between the PSFscorresponding to different capturing conditions like the PSF of thefirst capturing condition (h1, f1, F1, d1) and the PSF corresponding toa second capturing condition (h1, f2, F1, d1). Assume that f(x, y−a)denotes the PSF corresponding to the first capturing condition (h1, f1,F1, d1) and g(x, y−b) denotes the PSF corresponding to the secondcapturing condition (h1, f2, F1, d1). Then, the center-of-gravityposition of each PSF shifts in the y direction by “a” and “b” due to thelateral chromatic aberration component. The XY coordinate accords withthe xy coordinate of h(x, y) that is the PSF described with Expression(1).

Thus, the image processing/operating unit 22 moves the maximum intensitypositions or center-of-gravity positions of two PSFs to the coordinateorigin, and accords the center-of-gravity positions of the two PSFs witheach other. When the center-of-gravity positions of the PSFs areaccorded with one another, the primary components of the phases of theOTFs are made approximately equal.

Next, the image processing/operating unit 22 Fourier-transforms the PSFso as to converts it into the OTF, and stores it in the memory 24. Thenumber of OTFs corresponds to the number of capturing conditions. Asenclosed by solid lines in FIG. 2A, H1(u, v) denotes the OFTcorresponding to the first capturing condition, and H2(u, v) denotes theOFT corresponding to the second capturing condition. These OTFs arestored in the memory 24.

In obtaining H3(u, v) that is an OTF corresponding to the capturingcondition (h1, f3, F1, d1) of the image, the OTF corresponding to thefirst capturing condition (h1, f1, F1, d1) and the OTF corresponding tothe second capturing condition (h1, f2, F1, d1) are obtained from thememory 24 and weighted and added up.

For example, at an image height having a relationship of h1<h2<h3, OTFshaving image heights of h1 and h3 may be interpolated and an OTF havingan image height of h2 may be generated. When a focal length in zoominghas a relationship of f1≦f2≦f3, OTFs having focal lengths of f1 and f3may be interpolated and an OTF having a focal length of f2 may begenerated. In a diaphragm state having an F-value relationship ofF1≦F2≦F3, OTFs having diaphragm states of F1 and F3 may be interpolatedand an OTF having a diaphragm state of F2 may be generated. In an objectdistance of d1≦d2≦d3, OTFs having object distances of d1 and d3 may beinterpolated and an OTF having an object distance of d2 may begenerated.

As illustrated in FIG. 2B, the memory 24 stores PSF data in which themaximum intensity position or center-of-gravity position of the PSFshifts to the origin position, and may be used for the above processing.In other words, in FIG. 2B, two PSFs (f(x, y) and g(x, y)) each enclosedby a solid line is stored in the memory 24. A shift amount of the PSFmay be determined so that the RMS value of the PSF may be minimized foreach chromatic component.

For example, at an image height having a relationship of h1<h2<h3, PSFshaving image heights of h1 and h3 may be interpolated and a PSF havingan image height of h2 may be generated. When a focal length in zoominghas a relationship of f1≦f2≦f3, PSFs having focal lengths of f1 and f3may be interpolated and a PSF having a focal length of f2 may begenerated. In a diaphragm state having an F-value relationship ofF1≦F2≦F3, PSFs having diaphragm states of F1 and F3 may be interpolatedand a PSF having a diaphragm state of F2 may be generated. In an objectdistance of d1≦d2≦d3, PSFs having object distances of d1 and d3 may beinterpolated and a PSF having an object distance of d2 may be generated.

This is because an interpolation using the PSF is equivalent with aninterpolation using the OTF. One example is illustrated below:

$\begin{matrix}{\mspace{79mu}{{{h\left( {x,y} \right)} = \frac{{\left( {{f\; 3} - {f\; 2}} \right) \cdot {f\left( {x,y} \right)}} + {\left( {{f\; 2} - {f\; 1}} \right) \cdot {g\left( {x,y} \right)}}}{\left( {{f\; 2} - {f\; 1}} \right) + \left( {{f\; 3} - {f\; 2}} \right)}}{{{OTF}\left( {\mu,v} \right)} = {{{??}\left\{ {h\left( {x,y} \right)} \right\}} = \frac{{{\left( {{f\; 3} - {f\; 2}} \right) \cdot {??}}\left\{ {f\left( {x,y} \right)} \right\}} + {{\left( {{f\; 2} - {f\; 1}} \right) \cdot {??}}\left\{ {g\left( {x,y} \right)} \right\}}}{\left( {{f\; 2} - {f\; 1}} \right) + \left( {{f\; 3} - {f\; 2}} \right)}}}}} & (5)\end{matrix}$

In the meanwhile, h(x, y) denotes a PSF corresponding to an image pickupposition (h1, f2, F1, d1) after the interpolation is provided, andOTF(u, v) denotes that OTF.

It may be shifted so that the maximum intensity position of the PSFscorresponding to the first capturing condition (h1, f1, F1, d1) and thesecond capturing condition (h1, f2, F1, d1) or the root mean square(“RMS”) value can be minimum. Alternatively, it may be accorded with aposition other than the origin.

After the image restoration processing starts, the imageprocessing/operating unit 22 determines whether the capturing conditionof the captured image accords with one of the capturing conditionsstored in the memory 24 (S12). If so (Yes of S12), the imageprocessing/operating unit 22 produces an image restoring filter usingthe corresponding OTF data, and performs image restoration (S14).

On the other hand, when the OTF corresponding to the capturing conditionis not stored in the memory 24 (No of S12), the OTFs stored in thememory 24 are interpolated and an OTF is generated (S16), and the imageis restored with the generated OTF.

FIG. 3 is a view for explaining details of S16. Initially, asillustrated in FIG. 3A, the image pickup area is divided into N segmentsfrom the axial image height to the outermost off-axis image height, andOTFs are stored with the capturing conditions. For a region that has noOTF or capturing condition, two OTFs corresponding to two image heightpositions closest to a target position are weighted according to adistance and interpolated, and an OTF corresponding to the targetposition is generated.

Since the OTF of the image pickup optical system differs according tothe capturing condition, an OTF corresponding to a capturing conditionof capturing the object is generated through the interpolation based ondata of discretely existing, actual OTFs and capturing conditions. Forexample, the capturing condition may contain a focal length f of 20 mm,an F-value of 2.8, an object distance d of ∞, etc.

For simplicity, a description will be given of processing at an I-thimage height position. As illustrated by black dots in FIG. 3B, anactual capturing condition is located at a lattice point in athree-dimensional space with variables of a focal length, an F-value,and an object distance. A capturing condition with which correspondingOTF data actually exists on the three-dimensional space is illustratedby a black dot. A capturing condition (hI, fJ, FK, dL) under which animage is captured may be abbreviated by (I, j, k, l)=(I, J, K, L).

Initially, eight actual capturing conditions illustrated by black dotsin FIG. 3B are obtained: (i, j, k, l)=(I, 1, 1, 1), (I, 2, 1, 1), (I, 1,1, 2), (I, 2, 1, 2), (I, 1, 2, 1), (I, 2, 2, 1), (I, 1, 2, 2), (I, 2, 2,2). Corresponding OTFs are also acquired.

Next, as illustrated by black triangles in FIG. 3B, four firstinterpolation capturing conditions are prepared: An OTF corresponding to(I, J, 1, 1) is generated based on the two OTFs corresponding to (I, 1,1, 1) and (I, 2, 1, 1). An OTF corresponding to (I, J, 1, 2) isgenerated based on the two OTFs corresponding to (I, 1, 1, 2) and (I, 2,1, 2). An OTF corresponding to (I, J, 2, 1) is generated based on thetwo OTFs corresponding to (I, 1, 2, 1) and (I, 2, 2, 1). An OTFcorresponding to (I, J, 2, 2) is generated based on the two OTFscorresponding to (I, 1, 2, 2) and (I, 2, 2, 2).

Next, as illustrated by black rhombs in FIG. 3B, two secondinterpolation capturing conditions are prepared from the four firstinterpolation capturing condition: An OTF corresponding to (I, J, K, 1)is generated based on the two OTFs corresponding to (I, J, 1, 1) and (I,J, 2, 1), and an OTF corresponding to (I, J, K, 2) is generated based onthe two OTFs corresponding to (I, J, 1, 2) and (I, J, 2, 2).

Next, an OTF corresponding to a capturing condition (I, J, K, L) isgenerated based on the two OTFs corresponding to the two secondinterpolation capturing conditions (I, J, K, 1) and (I, J, K, 2). Thisembodiment thus provides interpolations by commonly using threevariables among the image height, the focal length, the F-value, and theobject distance.

The above example provides interpolations of the OTF data in order ofthe focal length, the F-value, and the object distance, but thepolarization order is not particularly limited to this order. While thisembodiment provides interpolation processing utilizing linear weighting,but a bi-cubic interpolation using a trigonometric function, or anothertype of interpolation may be used.

A similar approach may be used to obtain the OTF by interpolating thePSF in the real space area, and by performing a frequency conversion.

The image restoration processing in S14 and S18 use H3(u, v) of theinterpolated OTF for H(u, v) of Expressions (3) and (4). At this time,the interpolated OTF has no primary component in the phase, and cannotrestore the deterioration caused by the lateral chromatic aberration andthe distortion among the aberrations of the image pickup optical system.However, a deterioration caused by another aberrational component may berestored.

Accordingly, another embodiment restores an image by adding the primarycomponent of the phase of the OTF to an OTF generated through aninterpolation or an OTF generated by Fourier-transforming a PSFgenerated through an interpolation. Alternatively, as in S14 and S18 inthis embodiment, the image processing/operating unit 22 restores animage utilizing an OTF generated through an interpolation (which has noprimary component in the phase) or an OTF made by Fourier-transforming aPSF generated through an interpolation. Separate from the imagerestoration processing illustrated in FIG. 1B, known processing may beperformed for the restored image so as to reduce the lateral chromaticaberration or the distortion.

In FIG. 2A, the two PSFs are Fourier-transformed to generate the OTFswhile the center-of-gravity positions of the PSFs are accorded with eachother, and an OTF is interpolated by interpolating the secondary andhigher components of that phase. In addition, in FIG. 2B, while thecenter-of-gravity positions of the two PSFs are accorded with eachother, a phase component of the PSF corresponding to a capturingcondition of an image is obtained through the interpolation, and thenthe secondary and higher components of the phase of the correspondingOTF is generated through the Fourier transform.

The present invention is not limited to the embodiment illustrated inFIGS. 2A and 2B. For example, the image processing/operating unit 22Fourier-transforms the two PSFs, while their center-of-gravity positionsare not accorded with each other. Then, the image processing/operatingunit 22 eliminates the primary component of the phase from each of thetwo OTFs generated through the Fourier transforms, and generatessecondary and higher components of the phase an OTF corresponding to thecapturing condition of the image.

As described above, the image processing/operating unit 22 may generatethe OTF or PSF through the interpolation for each chromatic component ofthe RGB.

Second Embodiment

FIG. 4 is a block diagram of a digital camera (image pickup apparatus)according to a second embodiment. The digital camera includes an imagepickup optical system 401 that includes a diaphragm 401 a and a focusinglens 401 b and forms an optical image of an object. An image pickupelement 402 is configured to photoelectrically convert the optical imageinto an analogue electric signal. An A/D converter 403 converts theanalogue electric signal into a digital signal, and an image processor404 performs various image processing for the digital signal.

The various image processing contains the above image restorationprocessing. In other words, according to this embodiment, the imageprocessing apparatus is incorporated as an image processor 404 into thecamera. In this case, discrete OTF data (or PSF data) is stored in amemory 408. An image that has experienced various processing containingimage restoring is displayed on a display 405, or recorded in an imagerecording medium 409. Each component in the camera is controlled by asystem controller 410.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2011-148164, filed Jul. 4, 2011 which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: amemory configured to store information of an optical transfer functionor a point spread function of an image pickup optical system for atleast one of capturing conditions; and an image processor configured: togenerate secondary and higher components of a phase of an opticaltransfer function or a shape component of a point spread functioncorresponding to a capturing condition of an image captured through theimage pickup optical system through an interpolation based on at leasttwo optical transfer functions or point spread functions whichcorrespond to different capturing conditions and are derived from theinformation stored in the memory, while center-of-gravity positions ormaximum intensity positions are accorded with each other or differentialroot-mean-square values of the point spread functions are minimum, andto restore the image utilizing an optical transfer function derived fromthe optical transfer function that has been generated through theinterpolation or an optical transfer function derived from the pointspread function that has been generated through the interpolation. 2.The image processing apparatus according to claim 1, wherein theinformation stored in the memory is information of secondary and highercomponents from which a primary component is eliminated in the phase ofthe optical transfer function or the shape component of the point spreadfunction from which a center-of-gravity position component iseliminated.
 3. The image processing apparatus according to claim 1,wherein the information stored in the memory contains a primarycomponent of the phase of the optical transfer function or acenter-of-gravity position component of the point spread function, andwherein the image processor utilizes the information stored in thememory to generate pre-interpolation information of the secondary andhigher components from which the primary component is eliminated in thephase of the optical transfer function or the shape component from whicha center-of-gravity component of the point spread function iseliminated, and performs the interpolation utilizing thepre-interpolation information.
 4. The image processing apparatusaccording to claim 1, wherein the image processor restores the image byadding a primary component of the phase of the optical transfer functionto the optical transfer function generated through the interpolation orthe optical transfer function derived through a Fourier transform to thepoint spread function generated through the interpolation.
 5. The imageprocessing apparatus according to claim 1, wherein the image processorrestores the image utilizing the optical transfer function generatedthrough the interpolation or the optical transfer function derivedthrough a Fourier transform to the point spread function generatedthrough the interpolation, and performs processing for reducing alateral chromatic aberration or a distortion for a restored image. 6.The image processing apparatus according to claim 1, wherein the imageprocessor Fourier-transforms the two point spread functionscorresponding to the different capturing conditions while thecenter-of-gravity positions of the two point spread functions areaccorded with each other, and generates secondary and higher componentsof the phase of the optical transfer function corresponding to thecapturing condition of the image by interpolating the secondary andhigher components of the phase of the two optical transfer functionsgenerated through Fourier-transforming.
 7. The image processingapparatus according to claim 1, wherein the image processor obtains theshape component of the point spread function through the interpolationcorresponding to the capturing condition of the image whilecenter-of-gravity positions of two point spread functions correspondingto the different capturing condition are accorded with each other, andthen generates secondary and higher components of the phase of theoptical transfer function corresponding to the capturing condition ofthe image through Fourier-transforming.
 8. The image processingapparatus according to claim 1, wherein the image processor generatesthe optical transfer function or the point spread function through theinterpolation for each chromatic component of RGB.
 9. The imageprocessing apparatus according to claim 1, wherein at an image heighthaving a relationship of h1<h2<h3, the different capturing conditionsinclude image heights of h1 and h3, and the capturing condition of theimage has an image height of h2.
 10. The image processing apparatusaccording to claim 1, wherein when a focal length in zooming has arelationship of f1≦f2≦f3, the different capturing conditions includefocal lengths of f1 and f3, and the capturing condition of the image hasa focal length of f2.
 11. The image processing apparatus according toclaim 1, wherein in a diaphragm state having a relationship of F1≦F2≦F3,the different capturing conditions include diaphragm states of F1 andF3, and the capturing condition of the image has a diaphragm state ofF2.
 12. The image processing apparatus according to claim 1, wherein inan object distance of d1≦d2≦d3, the different capturing conditionsinclude object distances of d1 and d3, and the capturing condition ofthe image has an object distance of d2.
 13. The image processingapparatus according to claim 1, wherein the capturing conditions includean image height, a focal length, an F-value, and an object distance. 14.An image processing apparatus comprising: a memory configured to storeinformation of an optical transfer function or a point spread functionof an image pickup optical system for at least one of capturingconditions; and an image processor configured: to Fourier-transform twopoint spread functions corresponding to different capturing conditionswhile center-of-gravity positions of the two point spread functions arenot accorded with each other, to eliminate primary components of thephases from two optical transfer functions that are formed byFourier-transforming the two point spread functions, to generatesecondary and higher components of a phase of an optical transferfunction corresponding to a capturing condition of an image capturedthrough the image pickup optical system by interpolating the two opticaltransfer functions, and to restore the image utilizing the opticaltransfer function corresponding to the capturing condition.
 15. An imagepickup apparatus comprising: a memory configured to store information ofan optical transfer function or a point spread function of an imagepickup optical system for at least one of capturing conditions; and animage processor configured: to generate secondary and higher componentsof a phase of an optical transfer function or a shape component of apoint spread function corresponding to a capturing condition of an imagecaptured through the image pickup optical system through aninterpolation based on at least two optical transfer functions or pointspread functions which correspond to different capturing conditions andare derived from the information stored in the memory, whilecenter-of-gravity positions or maximum intensity positions are accordedwith each other or differential root-mean-square values of the pointspread functions are minimum, and to restore the image utilizing anoptical transfer function derived from the optical transfer functionthat has been generated through the interpolation or an optical transferfunction derived from the point spread function that has been generatedthrough the interpolation.
 16. A non-transitory computer-readablestorage medium storing a process for causing an information processingapparatus to execute a method comprising the steps of: storing, in amemory, information of an optical transfer function or a point spreadfunction of an image pickup optical system for at least one of capturingconditions; generating secondary and higher components of a phase of anoptical transfer function or a shape component of a point spreadfunction corresponding to a capturing condition of an image capturedthrough the image pickup optical system through an interpolation basedon at least two optical transfer functions or point spread functionswhich correspond to different capturing conditions and are derived fromthe information stored in the memory, while center-of-gravity positionsor maximum intensity positions are accorded with each other ordifferential root-mean-square values of the point spread functions areminimum; and restoring the image utilizing an optical transfer functionderived from the optical transfer function that has been generatedthrough the interpolation or an optical transfer function derived fromthe point spread function that has been generated through theinterpolation.